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PREFACE 


The  National  Science  Board  is  presently  exploring  how  the  U.S.  federal  government  sets  priorities  in 
research  and  development  and  whether  changes  are  needed  in  the  decision-making  process.  Accordingly,  the  NSB’s 
Committee  on  Strategic  Science  and  Engineering  Policy  Issues  asked  RAND’s  Science  and  Technology  Policy 
Institute  for  a  comprehensive  review  of  the  relevant  literature  and  experience  on  R&D  priority  setting  across  fields 
of  science.  The  resulting  report  surveys  the  literature  to  identify  descriptions  of  the  budget  coordination  and  priority 
setting  methodologies  currently  employed  by  the  federal  government  as  well  as  to  examine  critiques  of  currently 
employed  methodologies.  The  report  will  be  of  interest  to  those  with  general  interest  in  the  realm  of  science  and 
technology  policy  and  specifically  treats  issues  of  priority  setting  and  coordination  of  the  federal  R&D  portfolio 
across  fields  of  science. 


Created  in  Congress  in  1991,  the  Critical  Technologies  Institute  was  renamed  in  1998  as  the  Science  and 
Technology  Policy  Institute.  The  Institute  is  a  federally  funded  research  and  development  center  sponsored  by  the 
National  Science  Foundation  and  managed  by  RAND.  The  Institute’s  mission  is  to  help  improve  public  policy  by 
conducting  objective,  independent  research  and  analysis  on  policy  issues  that  involve  science  and  technology.  To 
this  end,  the  Institute 

•  supports  the  Office  of  Science  and  Technology  Policy  and  other  Executive  Branch  agencies,  offices, 
and  councils 

•  helps  science  and  technology  decisionmakers  understand  the  likely  consequences  of  their  decisions  and 
choose  among  alternative  policies 

•  improves  understanding  in  both  the  public  and  private  sectors  of  the  ways  in  which  science  and 
technology  can  better  serve  national  objectives. 

S&T  Policy  Institute  research  focuses  on  problems  of  science  and  technology  policy  that  involve  multiple 
agencies.  In  carrying  out  its  mission,  the  Institute  consults  broadly  with  representatives  from  private  industry, 
institutions  of  higher  education,  and  other  nonprofit  institutions. 

Inquiries  regarding  the  S&T  Policy  Institute  or  this  document  may  be  directed  to: 

Bruce  W.  Don 

Director,  Science  and  Technology  Policy  Institute 

1333  H  Street  NW 

Washington,  D.C.  20005 

Phone:  202-296-5000,  ext.  5351 

Web:  http://www.rand.org/centers/stpi 

Email:  stpi@rand.org 


EXECUTIVE  SUMMARY 


The  National  Science  Board  Committee  on  Strategic  Science  and  Engineering  Policy  Issues  asked  RAND  to  provide  a 
comprehensive  review  of  recent  literature  and  data  sources  on  priority  setting  and  coordination  in  federal  R&D.  The  full 
review  presents  a  synthesis  of  how  the  literature  describes  priority  setting  across  fields  of  science  and  the  issues  involved. 

We  have  identified  gaps  in  the  literature  where  the  process  remains  unclear  and  needs  explication.  We  conclude  with 
suggestions  for  further  study.  The  following  summary  presents  a  cursory  overview  of  the  main  points  in  the  review. 

OVERVIEW  OF  FINDINGS 

General  Assessment  of  the  Literature 

•  The  literature  is  weighted  toward  the  prescriptive  rather  than  the  descriptive  and  tends  to  take  a  broad  view 
rather  than  examine  operations  at  the  agency  level. 

•  There  is  a  robust  literature  offering  advice  to  government  on  how  best  to  set  goals  and  allocate  funds,  both  as  a 
national  endeavor  and  across  governmental  agencies,  falling  into  three  broad  categories: 

•  a  shift  to  a  national-goals  approach,  tying  the  priority  setting  process  to  national  goals; 

•  a  scientific-goals  approach  advocating  cross-cutting  assessments  of  existing  spending  in  areas  of  science  and 
realignment  of  budgets,  if  needed,  to  further  scientific  advancement; 

•  fine-tuning  of  the  existing  complex,  political  process. 

•  A  smaller  but  growing  base  of  procedural  publications  describes  how  the  process  of  R&D  allocation  should  be 
done  within  an  agency  or  a  discipline. 

•  Only  a  few  reports  describe  how  the  process  actually  takes  place  within  the  government  and  no  publications 
describe  the  process  across  fields  of  science. 

•  There  is  only  a  sparse  literature  describing  efforts  at  coordination. 

•  There  is  a  richer  discussion  of  goals  and  priority-setting  within  the  Executive  branch  than  within  Congress. 
Qualitative  discussions  of  how,  or  even  whether,  Congress  decides  among  funding  options  for  different  areas  of 
science,  different  federal  R&D  programs,  or  different  research  project  areas  are  comparatively  rare. 

•  Agencies  differ  in  setting  priorities  for  science  based  on  whether  they  have  a  scientific  or  mission  orientation. 
Most  agencies  now  gather  views  from  various  stakeholders  combined  with  strategic  planning  and  goal-setting. 

Key  Gaps  in  the  Literature 

While  high-level  goal  setting  is  discussed,  and  the  process  of  peer  review  and  scientific  advice  is  also  detailed,  there 
is  very  little  about  the  vast  middle  ground  where  goals-setting  meets  actual  funding  obligations.  Although  reports  cite  the 
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primary  role  of  the  Executive  Office  of  the  President  in  priority  setting  and  coordination,  relatively  little  exists  about  actual 
operations  such  as  the  role  of  the  NSTC  in  coordinating  federal  R&D.  The  literature  cites  NSTC  as  coordinating  larger 
initiatives  and  crosscuts,  although  the  importance  of  it’s  decisions  versus  those  of  OMB  staff  is  not  detailed  Likewise,  there 
is  little  description  of  the  NSTC  role,  if  any,  in  determining  funding  in  agency  core  R&D  programs  not  connected  to  a  larger 
budget  priority  or  “crosscut.” 

There  is  even  less  detail  about  the  process  that  takes  place  within  the  Executive  Branch  in  the  1 1  months  leading  up 
to  the  release  and  explanation  of  the  President’s  budget.  The  deliberation  within  agencies  for  resources  in  die  period  prior  to 
the  submission  of  the  proposed  budget  submission  to  OMB  in  September  is  nowhere  in  the  literature.  Likewise,  the  give- 
and-take  between  OMB  and  the  agencies  prior  to  the  agencies  being  “locked  out”  of  the  budget  in  December  is  not  described 
There  is  no  mention  of  how  the  individual  divisions  of  OMB  make  decisions,  set  priorities,  or  allocate  funding.  The 
readjustment  of  the  budget  that  occurs  after  the  agency  “lock  out”  in  December,  and  when  final  Presidential  priorities  are  set, 
is  not  described  in  the  literature. 

Furthermore,  there  is  little  description  of  the  ways  in  which  Congressional  committees  influence  the  direction  and 
conduct  of  federal  R&D  though  a  number  of  informal  means.  Rarer  still  are  documents  that  elaborate  on  either  the  details  of 
these  procedures  in  practice  or  the  degree  to  which  the  practice  corresponds  to  formal  procedures. 

Finally,  despite  the  sizable  academic  literature  on  methods  for  assessing  research  benefits,  there  is  virtually  no 
discussion  of  whether  or  how  these  have  been  implemented  by  the  research-sponsoring  community. 


Best  Practices  in  the  Literature 

The  literature  itself  offers  no  clear  concept  of  best  practice  nor  attempts  formally  to  make  such  an  assessment.  Doing 
so  would  require  establishing  a  metric,  a  task  difficult  to  perform  when  agency  missions  vary  so  greatly.  Yet,  the  literature 
might  be  said  to  imply  a  definition  of  best  practice  by  critiquing  present  practice,  as  discussed  below.  As  noted,  these 
critiques  generally  advocate  some  selective  change  rather  than  offer  an  integrated  design  and  might  be  said  more  to  offer 
views  of  “ideal”  practice  than  identify  best  practice. 

There  are  some  cases  where  the  U.S.  government  has  adopted  some  of  the  recommendations  made  in  different  reports 
but  the  effectiveness  of  these  changes  remains  unclear.  For  example,  the  White  House’s  creation  of  a  “21st  Century  Research 
Fund”  addresses  some  of  the  criticism  that  too  much  development  has  been  lumped  together  with  basic  research.  Responses 
to  the  increased  demand  for  accountability  of  science  and  technology  have  also  affected  priority  setting  practice  in  many 
R&D  agencies.  The  literature  has  yet  to  catch  up  with  these  developments,  but  these  changes  may  be  worth  further 
examination. 


Government- Wide  Coordination  and  Priority  Setting 

There  is  no  formally  defined  process  within  the  federal  government  to  set  goals  and  priorities  or  make  allocation 
decisions  for  science.  The  system  is  a  pluralistic  one  based  in  principle  on  promoting  excellence  and  relevance.  Many 
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players  with  different  interests  interact  to  influence  the  outcomes.  Recommendations  found  in  the  literature  on  setting  broad 
goals  for  federally  funded  research  fall  into  three  broad  categories: 

•  Tying  science  funding  more  tightly  to  national  goals; 

•  A  science  goals  approach  with  realignment  of  budgets,  if  needed,  to  better  meet  the  needs  of  scientific 
advancement;  and 

•  Fine  tuning  the  existing  complex,  political  process. 

Suggestions  for  more  detailed  models  of  priority  setting  in  turn  may  be  ascribed  to  three  categories: 

•  Engaging  the  scientific  community  in  determining  priorities  based  on  scientific  needs; 

•  Benchmarking  U.S.  capabilities  and  determining  where  more  emphasis  might  be  placed; 

•  Seeking  scientific  and  stakeholder  input  in  science  to  meet  agency  missions; 

Critiques  of  Currently  Employed  Methodology 

The  most  frequent  criticism  addresses  a  perceived  lack  of  clear  methodology  for  performing  priority  setting 
and  coordination.  Enactment  of  GPRA  has  led  to  changes  in  agencies’  practices,  yet  a  further  implicit  critique  may 
be  found  in  the  actions  of  the  House  Science  Committee  which  held  hearings  in  1 996  and  1 997  on  implementation 
in  the  civilian  science  agencies  and  announced,  in  1997,  that  this  would  be  a  major  oversight  target.  A  major 
argument  in  the  1995  NRC  “Press”  report  is  the  need  for  some  form  of  "comprehensive"  and  "coherent" 
coordination  of  federally-financed  research.  However,  even  this  recommendation  is  by  no  means  universally 
accepted. 


Advantages  and  Disadvantages  of  Different  Methodologies 

Best  practice  in  the  use  of  different  methodologies  suggest  that  a  pluralistic  approach  is  actually  the  more 
rational  way  to  make  determinations  among  competing  priorities.  For  example,  one  argument  against  a  more 
coherent  and  integrated  federal  S&T  budget  suggests  that  trade-offs  should  be  made  at  the  agency  level  between 
S&T  investment  and  other  expenditures;  the  Press  report  underestimates  the  value  of  the  mechanisms  already  in 
place,  especially  the  NSTC;  and  warns  against  the  “overly  comprehensive  process”  proposed  by  the  Press  panel. 

Other  voices  argue  that  the  budget  process  will  not  provide  a  method  or  even  an  analytic  framework  for 
setting  the  major  priorities  in  the  budget  because  of  the  diversity  of  agency  goals.  The  current  process  recognizes 
R&D’s  value  and  the  broad  acceptance  of  its  major  federal  role.  Yet,  it  is  too  difficult  to  budget  by  individual 
projects.  The  “level  of  effort”  approach  is  hard  to  defend,  especially  in  light  of  the  difficulty  of  making  causal 
arguments  by  tracking  direct  benefits.  Further,  under  current  practice,  the  fate  of  entire  disciplines  sometimes 
depends  upon  the  funding  decisions  of  individual  agencies. 
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Suggested  Approaches  to  Improving  Process 


Alternatives  to  the  present  processes  fall  into  one  of  three  areas:  alternative  weightings  or  other  means  for 
deriving  priorities  from  larger  national  goals;  suggestions  for  alternative  mechanisms  within  existing  institutions; 
and  changes  in  those  institutional  structures  themselves. 

Alternative  Weightings.  In  the  first  area,  there  are  calls  to  clarify  the  raison  d’etre  for  federal  R&D  support. 
There  are  frequent  recommendations,  for  example,  to  link  allocations  more  directly  to  specific  societal  goals. 
Whatever  criteria  are  chosen,  actual  processes  of  selection  and  allocation  should  be  more  explicit.  OTA  provided  an 
example  of  one  set  of  criteria  for  selecting  among  competing  initiatives  summarized  in  Table  S-l. 


Table  S-l.  OTA’s  Suggested  Criteria  for  Selecting  Among  Competing  R&D  Initiatives 


Scientific  Merit 

Scientific  objective  and  significance 

Breadth  of  interest 

Potential  for  new  discoveries  and  understanding 

Uniqueness 

Social  Benefits 

Contribution  to  scientific  awareness  or  improvement 
of  the  human  condition 

Contribution  to  international  understanding 

Contribution  to  national  pride  and  prestige 

Programmatic  Concerns 

Feasibility  and  readiness 

Scientific  logistics  and  infrastructure 

Community  commitment  and  readiness 

Institutional  implications 

International  involvement 

Cost  of  proposed  initiative 

In  addition  to  priorities  set  by  issue  area,  there  are  also  calls  to  do  so  by  stage  of  the  research  and  innovation 
process  or  other  criteria.  Similarly,  there  are  also  suggestions  to  shift  the  focus  of  funding  in  the  federal  R&D 
portfolio  dramatically  toward  basic  research  while  others  warn  that  parsing  the  federal  R&D  budget  by  the  old 
definitions  of  basic  and  applied  research  has  proved  politically  ineffective. 

Alternative  Mechanisms.  The  second  major  group  of  alternatives  addresses  the  mechanisms  by  which 
allocations  should  occur  within  existing  institutions.  The  concept  of  best  practice  might  be  applied  by  adapting 
already-existing  models  to  other  federal  agencies.  Several  suggestions  have  been  made  for  fundamental  changes  in 
allocation  methods.  One  would  use  an  options  approach  where  the  portfolio  is  constantly  updated,  balanced  for  risk, 
and  takes  advantage  of  increased  information  availability.  Many  view  the  current  system  as  largely  successful  for 
the  bulk  of  research  needs  but  suggest  that  within  a  pluralistic,  multi-agency  budgeting  approach,  some  areas  require 
special  attention  owing  to  their  large  potential  for  spillover  effects  to  other  agencies.  Several  reports  point  to  the 
paucity  of  data  gathering  and  the  necessity  for  establishing  a  database  of  the  federal  R&D  budget. 

Structural  Changes.  The  last  category  of  suggestions  addresses  the  institutions  of  federal  research  portfolio 
management  themselves.  Several  studies  advocate  a  greater  role  for  NSTC,  OSTP,  and/or  OMB  in  setting  portfolio 
guidelines.  This  would  constitute  a  fundamental  redrafting  of  the  role  of  these  agencies  and  the  nature  of  their 
interactions  with  the  rest  of  the  federal  research  portfolio  management  structure.  Improved  coordination  could 
require 

•  a  comprehensive,  comparable  data  base  on  R&D  budgets; 
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•  a  detailed  “directory”  report  to  provide  information  on  what  agencies  are  engaged  in  what  kinds  of 
R&D;  and 

•  a  report  on  “R&D  in  the  Budget”  each  year. 

Alongside  suggestions  for  different  goals  stands  the  suggestion  that  a  new  institutional  structure  be  created, 
such  as  a  non-governmental  National  Forum  on  S&T.  Such  a  body  might  also  define  what  the  essential  elements  in 
the  federal  R&D  portfolio  must  be  and  suggest  ways  in  which  the  portfolio’s  composition  may  be  more  readily 
adapted.  Some  proposals  call  for  creating  a  Federal  S&T  Budget  in  lieu  of  the  existing  post  hoc  accounting  concept 
and  also  shifting  from  a  bottom-up  to  a  top-down  process.  This  would  force  trade-offs  at  the  programmatic  level. 
Yet,  at  the  same  time  there  are  voices  stating  that  the  current  process  of  trade-offs  and  political  decisionmaking, 
influenced  by  advocates  of  science,  actually  works  fairly  well  and  meets  the  needs  of  science  for  adequate  funding. 


DEFINING  AND  DETAILING  "R&D"  AND  “S&T” 

Defining  R&D 

From  the  outset,  a  terminology  problem  confounds  attempts  to  characterize  the  literature  on  priority  setting  and  R&D. 
Although  the  terms  S&T  and  R&D  are  often  used  interchangeably,  they  have  very  different  meanings  in  the  context  of  the 
federal  government.  Specifically,  there  is  one  overarching  and  official  definition  of  R&D  used  by  all  federal  agencies1. 
Because  R&D  activities  constitute  die  primary  long-term  investment  of  die  federal  government  (education  and  training  is 
number  two),  R&D  is  separately  tracked  in  the  federal  budget.  2  Complicating  the  discussion  is  the  fact  that  other  terms 
have  been  introduced,  including  the  “Federal  Science  and  Technology  Budget”  and  the  “21s*  Century  Research  Fund.” 
Figure  S-l  shows  the  relationship  between  these  three  terms. 


1  R&D  is  a  budget  category  that  is  defined  by  the  Office  of  Management  and  Budget  (OMB)  in  Section  25  of  Circular  A-l  1. 

2  The  Department  of  Defense  (DOD)  alone  among  the  federal  agencies  has  refined  the  OMB  definition  by  sub-dividing  the 
Development  category.  DOD  then  takes  one  of  these  sub-categories,  groups  it  with  Basic  Research  and  Applied  Research,  and  collectively 
refers  to  these  three  activities  as  “S&T’  thus  designating  S&T  as  a  sub-set  of  R&D. 
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Figure  S-l:  The  National  Academy  of  Sciences  Comparison  of  3  “R&D”  Budgets 
FIGURE  2:  FY  2001  R&D,  FS&T,  and  21st  Century  Research  Fund  (in  billioas) 


Source:  Research  &  Development  FY20001.  Washington,  DC:  American  Association  for  the  Advancement  of  Science,  April  2000. 


Defining  S&T 

Parallel  to  the  designation  of  specific  federal  activities  as  R&D,  there  is  a  simultaneous  labeling  as  to  general 
purpose  (i.e.,  mission)  or  function  activities  for  S&T.  R&D  activities  are  found  in  every  functional  category  in  the 
federal  budget.  Of  particular  interest  are  the  R&D  activities  in  Function  250  -  General  Science,  Space  and 
Technology.  All  such  activities,  most  especially  those  of  NSF  and  much  of  DOE,  are  officially  labeled  as  S&T 
activities.  Only  a  part  of  the  activities  are  categorized  as  R&D.  Consequently,  for  these  agencies,  R&D  is  a  sub-set 
of  S&T. 

The  magnitude  of  civilian  agency  S&T  activities  is  hard  to  determine,  because  they  are  not  officially 
labeled  S&T.  Figure  2  illustrates  that  specific  activities  that  are  widely  believed  to  be  R&D  are  instead  S&T 
activities  that  fall  outside  the  set  of  activities  officially  designated  as  R&D  (e.g.,  the  Manufacturing  Extension 
Program  at  NIST  and  the  Space  Shuttle).  Failure  to  agree  on  the  definition  of  critical  terms  and  then  apply  them 
consistently  has  defeated  and  continues  to  defeat  basic  communication  in  the  federal  R&D  community. 

Data  Issues 

The  way  R&D  is  defined  affects  the  collection  and  sharing  of  government  data.  Data  on  the  contents  of  the 
federal  R&D  portfolio  contain  either  highly  aggregated  budget  information  or  disaggregated  project  descriptions. 
There  is  considerable  difficulty  finding  common  bases  for  combining  “crosscutting”  data  collected  by  different 
agencies.  Moreover,  activities  not  characterized  as  R&D  but  which  are  scientific  in  nature  (i.e.,  weather  data,  space 
travel,  mapping)  are  not  included  in  descriptions  of  federal  R&D  activities,  leading  to  some  confusion  during 
priority  setting  and  coordination  activities. 
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Figure  S-2  Contrasting  Defense-Related  and  Civilian  Definitions  of  S&T 


| _ R&D  | 

EXECUTIVE  BRANCH  ORGANIZATIONS 

The  actual  operation  and  effectiveness  of  executive  branch  organizations  and  processes  for  coordinating 
R&D  policy,  planning,  and  funding  are  poorly  described  in  the  literature.  Most  of  the  material  included  here  is 
derived  from  agency  procedural  documents. 

The  Executive  Office  of  the  President 

The  Executive  Office  of  the  President  has  four  offices  or  councils  that  advise  the  President  about  priority 
setting  in  R&D  and  S&T.  These  are: 

•  The  Office  of  Science  and  Technology  Policy.  OSTP  helps  coordinate  federal  science  activities  to  meet 
the  President’s  goals.  This  is  primarily  done,  in  the  Clinton  Administration,  through  the  National  Science 
and  Technology  Council  (NSTC)  for  which  OSTP  acts  as  a  secretariat.  OSTP,  together  with  the  Office  of 
Management  and  Budget,  issues  a  budget  memorandum  each  year  on  research  and  development  priorities. 

•  President’s  Council  of  Advisors  on  Science  and  Technology  (PCAST).  PCAST’s  principal  task  is  to 
assist  the  NSTC  in  securing  private  sector  involvement  in  the  latter’s  activities.  Some  of  its 
recommendations  are  general,  but  PCAST  also  makes  specific  recommendations  based  upon  assembled 
panels  as  well  as  its  own  reviews  of  reports  of  the  NSTC. 
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•  National  Science  and  Technology  Council  (NSTC)  was  established  to  integrate  the  President’s  S&T 
policy  agenda  across  the  federal  government  and  ensure  that  S&T  is  considered  in  development  and 
implementation  of  federal  policies  and  programs.  It  is  a  policy  and  budgetary  coordination  body  through 
which  all  executive  departments  and  agencies  coordinate  S&T  activities  that  require  significant  levels  of 
interagency  coordination.  OSTP  suggests  topics  around  which  the  NSTC  forms  committees  to  review 
government  spending  in  specific  areas  of  research  and  recommend  priority  or  allocation  shifts.  OSTP  then 
advises  agencies  and  OMB  and  solicits  input  from  the  larger  scientific  community  about  where  priorities 
and  resource  allocation  should  focus.  In  preparation  for  FY2001,  NSTC  is  overseeing  the  coordination  and 
priority-setting  for  1 1  areas  of  which  the  more  mature,  congressionally-mandated  programs  are  managed  as 
formal  interagency  crosscuts,  while  areas  being  developed  for  priority  attention  become  the  subject  of 
NSTC  working  groups. 

•  The  Office  of  Management  and  Budget  (OMB)  coordinates  the  President’s  budget  process.  This  process 
starts  each  summer  when  agencies  begin  preparing  their  budgets  for  the  fiscal  year  that  begins  in  October 
of  the  following  year.  There  is  no  “R&D  budget”  as  such.  OMB  has  the  power  to  shape  the  budget,  but 
does  not  set  specific  priorities  for  science.  It  does  examine  agency  proposals  for  redundancy  and  looks  for 
opportunities  for  interagency  coordination.  It  has  no  means  for  truly  setting  priorities  between  different 
R&D  programs  with  differing  goals.  Budget  guidelines  for  R&D  are  issued  jointly  by  OSTP  and  OMB. 
While  agency  officials  report  that  the  budget  call  does  have  some  effect  on  R&D  allocation,  it  may  actually 
affect  more  how  existing  plans  are  labeled  than  on  how  budget  priorities  are  set. 

The  priorities  for  R&D  that  become  guidelines  for  the  agency  budgeting  process  are  based  on  a  set  of  goals 
named  by  the  Clinton  Administration  in  the  first  months  of  its  tenure.  These  goals  include:  (1)  a  healthy,  educated 
citizenry,  (2)  job  creation  and  economic  growth,  (3)  world  leadership  in  science,  mathematics,  and  engineering,  (4) 
improved  environmental  quality;  (5)  harnessing  information  technology;  and  (6)  enhanced  national  security.  The 
R&D  priorities  have  remained  relatively  stable  over  the  past  six  years,  with  several  additions,  as  illustrated  in  Figure 
S-3  below. 
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Figure  S-3.  R&D  Priorities  Set  by  the  OSTP  and  OMB,  FY96-FY01 

FY96 _ FY97 _ FY98 _ FY99 _ FYOO _ FY01 

Partnership  for  a  new  generation  of  vehicle _ _ _ 

Biomedical  research,  health  promotion  and  disease  and  injury  prevention  research _ 

Learning  and  cognitive  processes  _ _ _ 

Materials  technology _ _ _ _ _ 

Energy  production  and  utilization  technologies _ _ _ 

Integrated  ecosystem  management _ _ _ 

Networking  and  communications _ _ 

Human-computer  interaction  _ _ _ 

Counter-terrorism _ _ _ 

Detection,  monitoring,  and  verification _ _ _ 

Civilian  aircraft _ _ _ 

Telemedicine 

Infectious  diseases _ _ _ 

Environmental  risk  assessment _ 

Microelectronics  _ . _ 

I  U.S.  Global  Climate  Change  Research 

I  Plant  genome  research 

I  Nanotechnology 


R&D  Agencies  Within  the  Executive  Branch 

The  literature  on  agency-level  R&D  priority  setting  is  not  a  robust  collection.  The  Office  of  Technology 
Assessment  report  “Federally  Funded  Research!  Decisions  for  a  Decade’  describes  these  activities,  and  this  section 
draws  heavily  from  that  report.  Beyond  this,  the  agencies  themselves  have  issued  GPRA-inspired  strategic  plans 
that  provide  some  insights  into  the  priority  setting  process.  Outside  of  these  sources,  we  found  very  little  concerning 
what  happens  in  the  agencies  with  regard  to  priority  setting,  despite  there  being  over  20  government  agencies 
funding  R&D.  It  makes  sense  that  the  largest  spenders  would  be  the  most  well  represented  in  the  literature,  but 
smaller  agencies  most  likely  make  dearer  trade-offs  in  funding.  These  smaller  agencies  may  be  worth  further 
examination.  Table  S-2  below  summarizes  what  exists  in  the  literature  about  agency  priority  setting  activities.  Not 
all  of  these  representations  may  be  current.  Some  of  the  literature  is  dated  and  changes  may  well  have  occurred  in 
these  agencies  since  the  original  report  was  written. 
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Table  S-2.  Agency  R&D  Priority  Setting  Activities 


AGENCY  ON  ORDER 
OF  THE  MAGNITUDE 
OF  THEIR  R&D 
BUDGETS) 

PRINCIPAL  FINDINGS  ABOUT  PRIORITY  SETTING 
ACTIVITIES  REPORTED  IN  LITERATURE 

METHODS  USED  TO  IDENTIFY 

PRIORITIES 

Department  of  Defense 

Planning  occurs  in  the  office  of  the  Director  of  Defense 
Research  and  Engineering  (DDR&E)  which  looks  to  the 

NSTC  and  the  Joint  Chiefs  of  Staffs  Joint  Vision  2010  for 
guidance. 

In  its  Basic  Research  Plan,  DOD  uses  peer  review 
and  competition  to  achieve  its  objectives; 

Technology  Area  Reviews  and  Assessments 
(TARA)  provide  an  oversight  function  to  assess 
the  quality  of  the  research  programs. 

Department  of  Energy 

A  National  Energy  Strategy  (NES)  was  designed  to  solicit 
input  from  the  offices  within  DOE  and  from  external  advisors. 
Each  program  has  an  advisory  panel,  such  as  the  High  Energy 
Physics  Advisory  Panel  (HEPAP)  and  the  Energy  Research 
Advisory  Board  (ERAB)  which  are  external  boards  of 
scientists.  These  groups  and  others  like  them  present  to  DOE 
a  set  of  priority  research  areas  that  deserve  the  agency’s 
special  attention. 

In  selecting  areas  of  research,  the  Office  of 

Science  emphasized  the  use  of  peer  review  to 
evaluate  all  programs.  It  also  stated  that  advisory 
boards  play  a  significant  part  in  its  priority  setting 
processes. 

NASA 

NASA  sets  priorities  in  conjunction  with  the  budget  process 
and  by  selecting  specific  projects.  Influenced  more  heavily  by 
Congress  than  other  agencies.  The  process  is  essentially 
bottom  up  with  project  managers  proposing  new  initiatives. 
When  large  missions  are  proposed,  such  as  Space  Station 
Freedom,  top-down  direction  determines  the  parameters  of  the 
effort.  (OTA) 

Priority  setting  results  from  a  combination  of 
input  from  NASA’s  own  internal  managers,  staff 
and  directors,  and  external  actors  like  the  National 
Research  Council  the  Task  Group  on  Space 
Astronomy  and  Astrophysics,  and  the  space 
science  community.  In  its  goal  to  pursue  scientific 
excellence,  the  Office  of  Science  emphasized  the 
use  of  peer  review  to  evaluate  all  programs. 

National  Institutes  of 

Health  (HHS) 

The  director  of  each  Institute,  with  the  help  of  NIH’s  national 
advisory  council,  decides  funding  direction  carried  out 
through  extramural  grants  and  intramural  programs. 

Advisory  councils  are  mandated  by  Congress  and 
composed  of  people  from  both  the  scientific 
community  and  the  public.  The  director  also 
consults  with  intramural  investigators  scientists  in 
the  extramural  program,  patients  and  their  families 
interested  in  research  on  particular  diseases, 
professional  and  scientific  groups,  representatives 
of  the  Administration  and  members  of  Congress, 
and  with  the  public. 

National  Science 

Foundation 

The  NSF  process  for  strategic  planning  involves  calling  in 
advisory  committees  and  committees  of  visitors,  regular 
reviews  of  programs  and  input  from  the  National  Science 

Board,  and  at  the  Directorate  level  reports  from  external 
groups  on  program  issues.  Goals  are  set  “by  scientific 
opportunity  and  the  proposal  process,  as  well  as  in  special 
initiatives  from  advisory  panels.” 

The  NSB  recommended  that  the  following  two 
criteria  be  adopted  in  place  of  the  four  criteria  that 
had  been  used  in  the  past  to  determine  research 
priorities: 

•  What  is  the  intellectual  merit  of  the  proposed 
activity?  E.g.,  does  it  advance  knowledge  and 
understanding  in  its  own  field  and  across  fields?  Is 
it  creative  and  original? 

•  What  are  the  broader  impacts  of  the 
proposed  activity?  E.g.,  advance  discovery  and 
promote  teaching?  Enhancing  partnerships? 

Department  of 

Agriculture 

USDA  derives  specific  priorities  from  its  1997  strategic  plan. 
Annual  performance  plans  are  modified  based  on  input  from 
the  staff  and  advisory  committees.  Priority  setting  is  advised 
by  many  groups,  most  important  is  the  Joint  Council  on  Food' 
and  Agriculture  Sciences  created  by  Congress. 

Budgets  are  developed  using  a  crosswalk  that 
links  the  strategic  goals  and  objectives  of  the 
agency  with  its  overall  budget  structure  and 
specific  performance  goals. 

Department  of 

Commerce,  National 
Institute  for  Standards  and 
Technology 

NIST  sets  priorities  in  specific  measurement  areas  based  on 
the  advice  of  councils  created  by  NIST  itself  but  which  are 
established  as  independent  nonprofit  organizations  as  well  as 
input  from  customers  and  NIST  scientific  and  technical  staff. 

The  councils  strive  to  provide  a  consensus  on 
industrial  and  academic  requirements  for 
standards  and  programs,  including  setting 
priorities.  Divisions  maintain  direct  contact  with 
customers  and  manufacturers  and  conduct 
periodic  customer  surveys  in  order  to  set  priorities 
based  on  customer  need. 

Environmental  Protection 
Agency 

EPA’s  Office  of  Research  and  Development  (ORD)  has  the 
principal  responsibility  for  research  and  development. 

Strategic  Plans  have  relied  heavily  on  EPA’s  Science 

Advisory  Board  (an  independent  group  of  engineering  and 
science  advisors)  and  expert  panels  convened  by  NAPA  and 
the  National  Academy  of  Sciences 

The  most  important  of  EPA’s  strategic  principles 
is  the  explicit  use  of  the  risk  paradigm  to  shape 
and  focus  EPA’s  organizational  structure  and 
research  agenda,  including  hazard  identification, 
dose  response  assessment,  exposure  assessment, 
and  risk  characterization. 
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CONGRESS 

Of  all  the  institutions  involved  in  coordination  and  priority  setting  across  fields  of  science,  the  literature  as 
a  whole  shows  its  largest  gap  in  its  treatment  of  Congress.  Congress  has  not  paralleled  the  Executive  branch  in 
coordinating  its  own  R&D  policy,  planning,  and  funding  efforts.  At  least  2 1  Congressional  Committees  have  direct 
federal  R&D  policy  or  funding  responsibility.  At  no  time  in  the  Congressional  process  is  there  a  comprehensive 
view  taken  of  the  R&D  portfolio  across  the  federal  government.  Further,  there  are  a  range  of  other  legislative 
decisions  that  can  affect  planning  and  priorities  of  federal  agencies  and  the  conduct  of  federally-funded  R&D. 
Regulatory,  tax,  or  other  decisions  affecting  research  institutions  are  made  outside  of  the  Congressional  circles  in 
which  R&D  policy  is  decided  and  are  frequently  not  coordinated  with  the  Congressional  entities  having  R&D  policy 
jurisdiction. 

INTERACTIONS  BETWEEN  THE  EXECUTIVE  AND  LEGISLATIVE  BRANCHES  ON  R&D 

The  need  for  interaction  between  branches  causes  the  shortcomings  found  within  each  to  become 
compounded  by  the  lack  of  a  formal  coordinating  mechanism.  Since  OMB  must  approve  agency  testimony,  any 
formal  presentation  to  Congress  will  serve  to  ratify  the  final  decisions  made  during  the  Budget  submission  and  will 
not  easily  provide  a  vehicle  for  an  R&D  agency  to  comment  critically  upon  decisions  made  in  that  process.  In 
addition,  agencies  are  generally  nonresponsive  to  questions  of  priority  put  to  them  by  Congress.  Yet,  oversight 
hearings  play  a  key  role  in  determining  the  budgets  of  specific  research  programs  and  encouraging  coordination 
between  the  research  agencies.  Absent  a  formal  agreement,  the  budget,  appropriations,  and  oversight  processes 
constitute  the  coordination  mechanisms  and  the  conduct  of  this  work  is  subject  to  individual  agency  and  committee 
dynamics  and  is  frequently  left  to  the  perspectives  and  proclivities  of  individual  members  of  Congress  and  then- 
staffs. 

USE  OF  BENEFITS  MEASURES  IN  PRIORITY  SETTING  AND  BUDGET  COORDINATION 

With  the  introduction  of  new  concepts  of  accountability,  the  R&D  agencies  have  begun  applying  benefits 
measures  to  R&D  and  using  the  results  to  help  set  priorities.  However,  this  process  has  not  been  studied  or 
systematically  documented.  It  may  be  too  early  in  the  process  of  adopting  these  measures  to  determine  if  they  are 
effective.  Most  of  the  measures  identified  in  the  literature  were  adopted  from  private  sector  applications.  In  many 
ways,  federal  research  presents  greater  problems  for  measurement  and  benchmarking  than  does  private  R&D.  A 
great  deal  of  federally  funded  research  is  directed  to  areas  where  the  market  is  limited  at  best.  Further,  given  the 
types  of  data  available,  the  returns  that  result  from  most  calculations  must  be  interpreted  as  average  rather  than 
marginal  rates.  From  a  policy  perspective,  this  means  we  cannot  be  certain  from  this  aggregate  analysis  what  the 
effect  of  an  additional  dollar  of  research  expenditure  might  be.  The  cost/benefit  framework  itself  may  be  too 
restrictive,  failing  to  capture  the  many  benefits  that  may  be  derived  from  publicly-funded  basic  research.  The  true 
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effect  of  such  outlays  may  well  be  indirect,  affecting  productivity  through  changing  the  returns  to  private  research 
and  development  rather  than  directly  as  a  result  of  the  specific  research  project 

Social  rates  of  return  analysis  seeks  to  determine  the  sum  of  benefits  accrued  from  changes  in  the  knowledge 
base  and  compares  these  benefits  to  the  cost  of  investment.  This  social  benefit  may  be  considerably  greater  than  the 
private  benefit  taken  in  the  form  of  profit.  As  a  practical  matter,  such  studies  involve  selecting  a  sample  of  specific 
innovations  upon  which  to  perform  these  calculations.  This  is  both  expensive  and  subject  to  unintentional  or 
unavoidable  bias  in  the  selection  process.  Further,  the  social  rate  of  return  calculated  by  such  means  is  not  directly 
comparable  to  the  internal  rates  of  return  calculated  for  private  investment  projects.  Nevertheless,  studies  in  this 
area  have  found  a  very  high  return  to  investment  in  basic  research. 

Among  potential  users  of  such  information  is,  of  course,  Congress.  Whether  it  acts  as  the  originator  of 
information  requests  for  the  purposes  of  furthering  its  own  process,  or  is  targeted  as  the  ultimate  audience  for 
assessments  produced  for  its  benefit  by  the  agencies  coming  before  its  committees  for  funding,  Congress  would  like 
to  have  better  means  for  determining  the  results  ensuing  from  federal  funding  of  R&D.  Another  body  which  has 
considered  broad  application  of  performance-based  measures  throughout  the  federal  government  is  the  NSTC. 

NSTC  has  issued  a  list  of  performance  measures  for  function  250  (R&D)  activities  that  encourage  setting  aside  80 
percent  or  more  of  R&D  for  peer-reviewed  competition  as  well  as  call  for  the  majority  of  assessments  to  be  made  by 
external  bodies.  These  are  in  reality  guidelines  for  conduct  and  measures  attuned  to  the  first  category  found  in  the 
literature:  asset-based  measures. 

The  National  Academy  of  Sciences  Committee  on  Science  Engineering  and  Public  Policy  (COSEPUP,) 
suggests  general  guidelines  for  measuring  U.S.  position  in  a  given  field  of  science:  1)  What  is  the  U.S.  position  in  a 
field?  2)  What  key  factors  determine  relative  position?  and  3)  What  is  the  trend  for  relative  position  in  the  near  and 
long  term?  These  yardsticks  do  not  necessarily  track  well  with  the  needs  of  mission  agencies.  Further,  in  practice, 
most  measures  appear  to  be  of  the  asset-accounting  type. 

At  the  agency  level,  the  DOE  Office  of  Science  Strategic  Plan  lists  a  series  of  success  indicators  for  each  of 
its  five  main  goal  areas  (e.g.:  “photochemical  systems  that  hold  promise  for  economical,  highly  efficient  solar 
cells.”)  The  indicators  are  outcome-oriented  but  seem  to  be  of  a  checklist-type,  attuned  to  achieving  particular 
milestones  and  not  quantitative  in  nature.  They  do  not  seek  to  track  direct  benefit  back  to  specific  R&D  project 
outcomes 

Documents  on  the  performance  assessment  process  in  use  by  NSF  point  to  heavy  reliance  on  external 
assessment.  Attempts  are  being  made  to  shift  from  a  somewhat  ad  hoc  basis  to  a  more  formal  procedure  that  will 
provide  a  common  format  to  the  review  process  that  extends  across  NSF.  For  example,  beginning  in  1998,  annual 
reports  to  the  Director  have  been  required  from  NSF  units  using  a  variety  of  indicators  and  data  series  to 
demonstrate  effect. 

Generally  speaking,  however,  although  new  advances  continue  to  be  suggested  in  the  academic  literature  and 
new  methodologies  for  identifying  and  selecting  new  research,  managing  existing  research,  and  evaluating  and 
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assessing  research  retrospectively  have  been  designed,  the  implementation  of  these  methods  by  the  research 
sponsoring  community  remains  minimal. 

CONCLUSIONS 

The  alternative  approaches  for  managing  the  federal  research  enterprise  that  emerge  from  the  literature  fall 
into  three  areas: 

•  alternative  readings  of  what  are  the  appropriate  goals  federal  research  support  should  seek  to  fulfill; 

•  suggestions  for  alternative  mechanisms  for  allocation  of  funds  within  the  existing  institutions  for 
managing  the  federal  research  portfolio;  and 

•  elements  for  a  design  leading  in  whole  or  in  part  to  changes  in  those  institutional  structures  themselves. 
The  two  consistent  themes  are  a  desire  to  establish  priorities  and  to  do  so  in  a  coordinated  fashion.  This  assumes 
that  improvements  in  either  the  top-down  or  the  bottom-up  approach  or  both  would  improve  the  outcome.  This 
assumption  has  not  been  questioned  in  the  literature. 

This  is  understandable.  A  frequent  theme  of  the  literature  is  that  the  federal  R&D  portfolio  is  only  a  post 
facto  accounting  concept.  It  is,  by  default,  the  aggregation  of  individual  mission  agency  portfolios  but  is  in  no  sense 
managed  ex  ante  as  a  unified  portfolio.  Several  studies  advocate  a  greater  role  for  the  NSTC,  OSTP,  and/or  OMB  in 
setting  portfolio  guidelines  at  a  higher  level  than  that  of  the  funding  agencies  as  well  as  actively  monitoring 
fulfillment.  A  necessary  first  step  to  effective  prioritization,  in  other  words,  is  to  achieve  coordination. 

Nevertheless,  the  issue  of  whether  a  more  unified  or  better  coordinated  portfolio  is  desireable  or  achieveable  has  not 
been  adequately  debated  in  the  literature  and  deserves  more  attention. 

A  persistent  assumption  in  the  literature  is  that  greater  coordination  is  desirable  and  can  be  attained  by 
setting  high  level  goals  and  then  proceeding  to  lower  levels  of  decisionmaking  authority.  This  seems  problematic  on 
two  counts.  First,  decisionmaking  in  this  area  is  embedded  in  existing  institutions  and  political  processes.  Setting 
high-level  goals  and  then  rigorously  enforcing  them  as  the  means  for  crafting  priorities  and  making  allocations  on 
lower  levels  would,  in  effect,  stand  the  present  system  on  its  head. 

Second,  such  an  approach  may  not  accord  with  the  evolving  pattern  for  the  commerce  of  ideas  and 
knowledge  management.  Modem  science  is  increasingly  cross-disciplinary,  with  major  discovery  taking  place  at 
the  interstices  of  traditional  disciplinary  categories.  Contributing  to  this  trend  are  increased  globalization  of  effort 
with  geographically  dispersed  working  teams  crossing  geographic  boundaries  and  an  ever  denser  connectivity  of 
information  and  ideas. 

This  is  not  to  suggest  that  prioritization  or  coordination  are  undesirable  or  that  gaining  a  measure  of  control 
is  impossible.  Rather,  it  is  to  suggest  that  as  we  proceed  further  along  this  line  of  inquiry  we  should  address  the 
following  questions  left  largely  unadressed  by  the  literature  as  it  stands: 

1 .  What  do  we  mean  by  the  terms  “priority”  and  “coordination”? 

2.  What  do  we  hope  to  achieve  and  how  will  we  improve  the  public’s  lot  through  prioritization  and 
coordination? 

3.  What  are  appropriate  measures  for  identifying  best  practice  in  priority  setting  and  coordination? 
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4.  What  alternative  models,  not  necessarily  predicated  upon  traditional  views  of  either  the  science  process 
or  its  effect  on  the  larger  society,  need  we  consider  to  best  develop  means  for  achieving  a  true  ability  to 
set  priority  and  the  level  of  coordination  we  desire? 

In  order  to  fully  understand  the  processes  that  take  place  within  the  system  that  result  in  the  set  of  activities 
that  the  government  labels  “R&D”  or  “S&T”  the  Board  needs  a  better  understanding  of  what  is  happening  in  the 
agencies  or  in  the  scientific  community  in  that  “vast  middle”  between  high  level  goals  and  bottom-up  input.  The 
decisions  made  at  the  program  and  project  level  have  not  been  studied  or  described  in  the  literature.  Insights  that 
could  be  gained  from  an  examination  of  these  activities  may  aid  the  Board  in  its  effort  to  bring  more  accountability 
and  coordiantion  to  the  process. 
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materials  science  and  engineering,  and  immunology.  The  agencies  submitted  strategic  plans  to  the  Congress 
in  September  1997  and  delivered  annual  performance  plans  with  FY1999  budget  justifications. 

—  National  Research  Council,  Harnessing  Science  and  Technology  for  America ’s  Economic  Future:  National  And 
Regional  Priorities.  Washington  DC:  National  Academy  Press,  1999. 

Report  makes  recommendations  to  develop  new  mechanisms  for  international  research  collaboration 
to  advance  fundamental  knowledge,  drawing  on  the  experience  of  recent  years.  After  the  federal 
government,  the  academic  institutions  performing  research  and  development  (R&D)  provided  the 
second  largest  share  of  academic  R&D  support.  The  NRC  report  noted  that  much  of  this  funding 
comes  from  state  governments,  but  is  counted  as  institutional  funding  because  the  university  has 
discretion  over  whether  it  will  be  spent  on  research  or  in  other  ways.  Industrial  R&D  support  for 
academic  institutions  has  grown  more  rapidly  than  support  from  other  sources  since  1980  (i.e.,  in 
constant  dollars,  industrial-financed  R&D  increased  by  an  estimated  250%  from  1980  to  1995,  and 
industry’s  share  grew  from  3.9%  to  6.9%)  (NRC,  1999).  More  extensive  university-industry 
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collaboration  on  long-term  issues  of  interest  to  industry  could  help  to  alleviate  the  funding  pressures 
being  faced  by  universities  (NRC,  1999). 

—  National  Research  Council.  An  Assessment  of  the  National  Institute  of  Standards  and  Technology  Measurement 

and  Standards  Laboratories.  Washington  DC:  National  Academy  Press,  1998. 

Each  year  since  1959,  the  National  Research  Council  has  assessed  the  programs  of  the  National 
Institute  of  Standards  and  Technology  (NIST),  and  its  predecessor,  the  National  Bureau  of  Standards. 
Assessments  are  currently  performed  by  about  150  leading  scientists  and  engineers,  equally  from 
U.S.  industry  and  academe,  appointed  by  the  National  Research  Council  (NRC),  and  administered  by 
the  NRC's  Board  on  Assessment  of  NIST  Programs.  There  are  currently  seven  major  Panels  that 
assess  the  major  organizational  areas:  electronics  and  electrical  engineering,  manufacturing 
engineering,  chemical  science  and  technology,  physics,  materials  science  and  engineering,  building 
and  fire  research,  and  information  technology. 

—  National  Research  Council.  “International  Benchmarking  of  US  Materials  Science  and  Engineering  Research,” 
Committee  on  Science,  Engineering  and  Public  Policy.  Washington,  DC:  National  Academies  Press,  1998. 

—  Observations  on  the  President's  Fiscal  Year  1999  Federal  Science  and  Technology  Budget,  Washington, 

DC:  National  Academies  Press,  April  1998. 

—  Institute  of  Medicine,  Scientific  Opportunities  and  Public  Needs:  Improving  Priority  Setting  and  Public  Input  at 
the  National  Institute  of  Health,  Washington,  D.C.:  National  Academy  Press,  1998. 

This  report  examines  the  way  in  which  NIH  sets  priorities  and  provides  a  few  recommendations  for 
improvement.  The  report  states  that  NIH’s  objectives  should  revolve  around  identifying  the  public's 
health  needs,  extending  basic  research  and.  The  report  recommends  that  NIH  continue  to  use  its 
current  method  for  criteria  setting,  but  implement  a  more  systematic  use  and  analysis  of  data  sources 
for  input  in  priority  setting  .  The  report  also  recommends  an  increased  role  for  NIH’s  Advisory 
Committee  as  well  as  the  establishment  of  a  Public  Liaison  Office. 

—  National  Research  Council,  A  New  Science  Strategy  for  Space  Astronomy  and  Astrophysics,  Washington,  D.C.: 
National  Academy  Press,  1997. 

This  update  to  the  1991  report.  The  Decade  of  Discovery  in  Astronomy  and  Astrophysics,  uses 
priority-setting  methods  established  in  1991  to  provide  a  strategy  for  space  astronomy  and 
astrophysics.  In  doing  its  priority  setting,  the  Task  Group  on  Space  Astronomy  and  Astrophysics,  the 
community  (1)  concentrated  on  the  scientific  objectives  rather  than  the  method;  (2)  prioritized 
scientific  questions  according  to  whole  classes  of  astronomical  objects,  rather  than  to  individual 
observing  bands;  and  (3)  looked  realistically  at  cost  and  technical  feasibility. 

—  National  Research  Council.  Understanding  Risk:  Informing  Decisions  in  a  Democratic  Society.  Washington, 

D.C:  National  Academy  Press,  1996. 

—  Board  on  Environmental  Studies  and  Toxicology.  “Interim  Report  of  the  Committee  on  Research  and  Peer 
Review  at  EPA.”  Washington,  D.C.  National  Academy  Press,  1995. 

—  Committee  on  Criteria  for  Federal  Support  of  Research  and  Development,  Allocating  Federal  Funds  for  Science 
and  Technology,  Washington,  D.C.:  National  Academy  Press,  1995. 

The  Committee  on  Criteria  for  Federal  Support  of  R&D  provides  an  overview  of  how  R&D  is 
defined  within  the  federal  government  and  describes  the  current  process  of  allocating  R&D  funds 
through  federal  departments  and  agencies.  Based  on  this  information  and  a  literature  review,  the 
committee  recommends  three  policy  initiatives  for  allocating  federal  funds:  (1)  The  President  should 
present  an  annual  comprehensive  FS&T  budget;  (2)  the  departments  and  agencies  should  make  FS&T 
allocation  decisions  based  on  clearly  articulated  criteria  that  are  congruent  with  those  that  the 
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Executive  Office  of  the  President  and  Congress  use;  and  (3)  Congress  should  create  a  process  that 
examines  the  entire  Federal  Science  &  Technology  budget  before  the  total  federal  budget  is 
disaggregated. 


—  National  Research  Council.  Science  and  Judgement  in  Risk  Assessment.  Washington,  D.C.:  National  Academy 

Press,  1994. 

—  Science,  Technology,  and  the  Federal  Government:  National  Goals  for  a  New  Era.  Washington  D.C.:  National 
Academy  Press,  1993. 

This  report  recommends  tying  S&T  goals  to  two  overarching  principles:  (1)  The  U.S.  should  be 
among  the  world  leaders  in  all  major  areas  of  S&T,  and  (2)  the  U.S.  should  maintain  clear  dominance 
in  scientific  fields  likely  to  contribute  to  substantially  important  economic,  social,  or  cultural 
objectives.  Further,  government  should  cooperate  with  the  private  sector  to  maintain  U.S.  leadership 
in  technologies  that  promise  to  have  major  influence  on  industrial  and  economic  performance  and 
that  could  lead  to  new  industries,  based  on  principles  of  cost-sharing,  insulation  from  distributional 
politics,  and  stable  support. 


—  National  Research  Council,  Astronomy  and  Astrophysics  Survey  Committee,  The  Decade  of  Discovery  in 
Astronomy  and  Astrophysics.  Washington,  D.C.:  National  Academy  Press,  1991. 

This  report  discusses  the  results  of  a  survey  conducted  by  the  Astronomy  and  Astrophysics  Survey 
Committee  of  the  NRC.  The  study  was  commissioned  to  provide  an  overview  of  what  is  going  on  in 
astronomy  and  to  recommend  initiatives  for  the  coming  decade.  The  committee  was  tasked  to 
provide  a  prioritized  list  of  instruments  for  the  coming  decade,  evaluate  the  existing  infrastructure, 
explore  the  consequences  of  the  computer  revolution  for  astronomy,  prepare  a  popular  summary  of 
opportunities  for  scientific  advances  in  astronomy,  and  suggest  possible  areas  for  developing  new 
observational  technologies. 

—  Presidents  of  the  National  Academy  of  Sciences,  National  Academy  of  Engineering,  and  Institute  of  Medicine, 
Federal  Science  and  Technology  Budget  Priorities.  Washington,  D.C.:  National  Academy  Press,  1988. 

This  report  notes  that  the  absence  of  a  coordinated  national  R&D  budget  and  lack  of  suitable  criteria 
for  making  global  R&D  budget  decisions  hinders  effective  use  of  federal  dollars.  The  report  notes 
that  priority-setting  within  agency  missions  is  adequate  and  that  a  pluralistic  approach  to  budgeting 
has  been  a  strength  of  the  U.S.  system.  In  three  classes  of  activity,  however,  special  attention  is 
needed:  (1)  initiatives  contributing  to  the  science  base,  (2)  initiatives  tied  to  presidential  or 
congressional  directives,  and  (3)  major  “megascience”  projects  slated  for  rapid  growth  or  large  pieces 
of  the  budget. 

— Risk  Assessment  in  the  Federal  Government:  Managing  the  Process.  Washington,  D.C.:  National  Academy  Press 
1983. 

National  Institutes  of  Health.  Setting  Priorities  at  the  National  Institutes  of  Health  .  Washington,  D.C.:  Department 
of  Health  and  Human  Services,  September  1997. 

NIH’s  Working  Group  on  Priority  Setting  provides  a  description  of  the  way  that  the  NIH  set’s 
priorities.  According  to  the  Group,  the  NIH  provides  funding  to  programs  by  1)  responding  to  public 
health  needs  2)  following  a  stringent  peer  review  system  and  3)  diversifying  its  research  portfolio  to 
include  a  variety  of  research.  Input  into  which  research  programs  NIH  will  pursue  depends  on  a  the 
advise  of  a  variety  of  actors  from  the  extramural  science  community  to  Congress  and  the 
Administration. 

National  Institutes  of  Health.  NCRR.  A  Catalyst  for  Discovery.  A  Plan  for  the  National  Center  for  Research 
Resources  1998-2003.  Bethesda,  MD:  Office  of  Science  Policy,  NCRR/NIH,  1998. 
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National  Science  Board,  “Government  Funding  of  Scientific  Research,”  working  paper,  Washington,  D.C.:  U.S. 
Government  Printing  Office,  NSB-97-186,  1997. 

The  NSB  report  calls  for  mandatory  priority  setting  and  coordination  of  federal  R&D.  Report 
provides  a  follow  up  to  its  1997  announcement  that  NSB  would  play  a  larger  role  in  setting  national 
S&T  priorities  and  policy.  Separate  House  and  Senate  science  policy  efforts  are  also  described  under 
the  FY1999  budget  section. 

_  “Overview:  Science  and  Technology  in  Transition  to  the  21st  Century,”  in  Science  and  Engineering  Indicators, 
Washington,  D.C.:  National  Science  Foundation,  1998. 

National  Aeronautics  and  Space  Administration.  “NASA  Performance  Plan  Fiscal  Year  1999.”  Washington,  DC: 
U.S.  Government  Printing  Office,  February  1999. 

This  GPRA  document  describes  performance  measures  for  FY99  Budget  activities. 


National  Science  Foundation.  “FY1999  GPRA  Performance  Plan”,  Washington,  DC:  National  Science  Foundation, 
January  1999. 

—  GPRA  Strategic  Plan  FY1997-FY2003.  Washington,  DC:  National  Science  Foundation,  September  1997. 

National  Science  and  Technology  Council.  National  Nanotechnology  Initiative:  Leading  to  the  Next  Industrial 
Revolution.  February  2000. 

~  Strategic  Planning  Document  -  Transportation  R&D.  National  Science  and  Technology  Council  (NSTC) 

1995  Strategic  Planning  Documents,  March,  1995. 

This  summary  report  of  Federal  transportation  research  and  development  priorities  was  prepared  for 
the  National  Science  and  Technology  Council  (NSTC)  by  the  NSTC  Interagency  Coordinating 
Committee  on  Transportation  R&D  and  the  Office  of  Science  and  Technology  Policy.  The  strategic 
plan  reflects  the  initial  efforts  of  the  Committee  to  assess  Federal  research  and  to  develop  long-term 
R&D  programs  integrated  across  agencies  in  specific  transportation-related  areas  of  common  interest. 

It  is  based  primarily  on  materials  developed  by  the  subcommittees  and  working  groups,  working 
within  the  framework  established  by  the  full  committee  in  its  Strategic  Budget  Guidance  report 
presented  to  NSTC  in  April,  1994. 

The  summary  report  was  compiled  from  subcommittee  submissions  by  staff  of  DOT'S  Volpe  National 
Transportation  Systems  Center  under  the  direction  of  Noah  Rifkin,  Executive  Secretary  of  the 
Committee  and  DOT  Director  of  Technology  Deployment  and  by  the  White  House  Office  of  Science 
and  Technology  Policy.  The  subcommittee  report  contains  extensive  additional  detail  concerning 
agency  programs,  goals,  issues  and  resources.  Efforts  of  the  Committee  in  1994,  summarized  in  this 
document,  focused  on  identification  of  perceived  R&D  gaps  and  opportunities.  They  provide  the 
foundation  for  generation  in  1995  of  a  detailed  and  comprehensive  description  of  Federal 
transportation  R&D  goals,  plans,  measures,  budgets  and  priorities,  including  active  coordination  with 
other  NSTC  Committees. 

—1998  Annual  Report,  Washington,  D.C.:  National  Science  and  Technology  Council  Program  Office,  1998. 

—1997  Annual  Report,  Washington,  D.C.:  National  Science  and  Technology  Council  Program  Office,  1997. 

Office  of  Technology  Assessment,  Federally  Funded  Research:  Decisions  for  a  Decade,  Washington,  D.C.:  U.S. 
Government  Printing  Office,  OTA-SET-490,  May  1991. 

This  study  suggests  that  the  criteria  used  to  set  priorities  for  various  areas  of  research  lack  explicit 
guidelines,  particularly  at  the  highest  levels  of  allocation,  leading  to  widely  varying  criteria  and 
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outcomes.  OTA  also  commented  that  the  lack  of  a  mechanism  for  evaluating  the  total  research 
portfolio  of  the  federal  government  in  terms  of  progress  toward  many  national  objectives  results  in 
S&T  being  only  loosely  tied  to  needs.  Finally,  the  federal  S&T  enterprise  should  seek  to  include 
criteria  beyond  scientific  merit  and  mission  relevance  when  judging  the  worth  of  a  research  program. 

The  report  calls  for  OSTP  to  disclose  the  criteria  by  which  federal  S&T  priorities  are  set. 

President’s  Committee  of  Advisors  on  Science  and  Technology,  Review  of  the  Proposed  National  Nanotechnology 
Initiative,  November  1999. 

—  Letter  to  President,  6  December  1996. 

—  PCAST  Fusion  Review  Panel,  The  U.S.  Program  of  Fusion  Energy  Research  and  Development,  1 1  July  1995. 

Popper,  Steven  W.,  “Policy  Perspectives  on  Measuring  the  Economic  and  Social  Benefits  of  Fundamental  Science,” 
RAND  MR-1 130-STPI.  Santa  Monica,  CA:  RAND,  September  1999. 

Press,  Frank,  “Criteria  for  the  Choice  of  Federal  Support,”  AAAS  Science  and  Technology  Policy  Yearbook 
1996/1997,  Washington  D.C.:  American  Association  for  the  Advancement  of  Science,  1997,  pp.  171-178. 

Frank  Press,  chair  of  the  committee  that  published  a  report  entitled  Allocating  Federal  Funds  for 
Science  and  Technology ,  builds  upon  and  reacts  to  ideas  put  forth  in  the  report.  The  articles  states 
that  we  need  to  make  the  idea  of  a  federal  science  and  technology  budget  a  reality,  one  that  not  only 
contains  budget  numbers  and  definitions  but  also  provides  a  process  for  upgrading  the  S&T  portfolios 
of  agencies  by  forcing  trade-offs.  Appropriations  for  this  budget  can  be  debated  within  a  new 
subcommittee  created  for  the  specific  puipose  of  evaluating  the  FS&T  budget.  This  type  of  structure, 
however,  has  come  under  criticism  for  several  reasons  (1)  it  might  make  the  FS&T  budget  vulnerable 
during  times  of  budget  deficits,  (2)  it  may  result  in  a  decrease  in  the  overall  budget  pool  for  S&T,  and 
(3)  it  may  create  conflict  within  the  science  community  to  increase  the  budget  instead  of  complying 
with  constrictions,  and  (4)  because  the  NAS  report  is  itself  viewed  biased  in  favor  of  federal  labs  and 
universities. 


Robinson,  David  Z.,  “Think  Twice  before  Overhauling  Federal  Budgeting  Science  and  Technology  Policy 

Yearbook  1996/1997,  Washington  D.C.:  American  Association  for  the  Advancement  of  Science,  1997,  pp. 
217-224. 

This  article  disagrees  with  the  ideas  proposed  in  the  NAS  report,  Allocating  Federal  Funds  for 
Science  and  Technology ,  calling  an  FS&T  budget  structure  conceptually  and  practically  wrong. 

Robinson  continues  to  state  that  trade-offs  should  not  be  made  between  categories  of  FS&T 
investment  but  between  S&T  and  other  expenditures  within  a  federal  agency.  He  continues  to  argue 
that  mechanisms  are  already  in  place  to  review  specific  areas  of  R&D  duplication.  Robinson 
recommends  that,  instead,  policymakers  should  determine  the  appropriate  level  of  support  by  linking 
FS&T  programs  to  national  goals  while  making  trade-offs  between  current  and  future  needs. 


Saunders,  Kenneth  V.  et  ah,  Priority-Setting  and  Strategic  Sourcing  in  the  Naval  Research,  Development,  and 
Technology  Infrastructure.  Santa  Monica,  CA:  RAND,  1995. 

This  report  suggests  ways  in  which  the  Dept,  of  the  Navy  might  realize  more  value  from  its 
increasingly  constrained  research,  development,  and  technology  (RD&T)  dollars.  The  study  was 
motivated  by  the  Navy’s  immediate  policy  needs  in  connection  with  the  1995  round  of  Base 
Realignment  and  Closure  (BRAC)  and  its  longer-term  need  to  make  the  best  use  of  its  resources. 
Suggestions  are  presented  in  three  parts.  First,  the  authors  develop  and  apply  a  framework  for  setting 
funding  priorities  in  the  Naval  RD&T  infrastructure.  Second,  the  authors  discuss  alternative  RD&T 
procurement  arrangements  that  are  seeing  increasing  use  in  the  private  sector  and  that  have  been  used 
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in  various  parts  of  the  government.  These  are  commonly  called  "smart  buying,"  but  the  authors  use 
the  term"  strategic  sourcing."  Third,  the  authors  present  a  speculative  combination  of  the  priority¬ 
setting  and  strategic-sourcing  considerations  of  the  first  two  parts.  Using  a  reinterpretation  of  the 
orthogonal  plot  developed  earlier  in  the  report,  it  suggests  a  way  to  help  determine  which  parts  of  the 
Naval  RD&T  infrastructure  are  best  suited  for  alternative  procurement  arrangements.  It  also  suggests 
a  way  to  determine  which  facilities  might  be  involved. 

Science  and  Government  Report,  Mar.  1, 1997. 

The  Vice  President  for  Research  at  the  University  of  Michigan,  proposed  a  high-level 
public/govemmental  commission  to  assess  "'the  rationale  for  investments  in  research’ 
by.. .governments,  industry  and  universities.. .the  division  of  labor  among  academic,  industrial  and 
government  laboratories;  criteria  for  setting  levels  of  R&D  support,  and  the  implications  of  current 
long-term  spending  projections  for  research.” 

Shapley,  Willis  H.,  The  Budget  Process  and  R&D.  Washington,  D.C.:  Carnegie  Commission  on  Science, 
Technology,  and  Government,  April  1992. 

This  report  describes  and  discusses  the  federal  budget  process  with  a  focus  on  R&D.  Shapley 
proposes  and  addresses  several  concerns  that  have  arisen  in  creating  a  federal  R&D,  including  (1) 
setting  priorities  and  achieving  balance,  (2)  the  use  of  budget  data,  (3)  the  stability  and  continuity  of 
the  current  budget  process,  and  (4)  the  fragmentation  of  R&D  in  the  budget  review  process  in  OMB 
and  Congress.  Shapley  states  that  (1)  R&D  funding  for  programs  should  not  be  pitted  against  each 
other  but  rather  against  the  overall  federal  budget;  (2)  a  comprehensive  comparable  databank  on 
R&D  budgets  should  be  established,  as  proposed  in  a  1988  report  of  the  Senate  Budget  Committee; 
(3)  a  partial  rather  than  immediate  implementation  of  a  two-year  appropriation  cycle  is  more 
politically  saleable  and  (4)  subcommittee  hearings  for  R&D  should  not  be  done  by  a  separate 
committee,  because  this  could  make  R&D  agencies  more  vulnerable  to  arbitrary  reductions.  Finally, 
Shapley  states  that  there  is  a  significant  shortfall  in  the  R&D  budget  in  meeting  important  needs  and 
grasping  important  opportunities.  The  report  states  that  the  nature  of  R&D  makes  it  a  necessity  to 
increase  funding  in  certain  S&T  areas  to  keep  pace  with  advances;  however,  this  is  difficult  to 
accomplish  in  a  deficit-ridden  budget. 


Smith,  Philip  M.,  and  Michael  McGeary,  “Don’t  Look  Back:  Science  Funding  for  the  Future,”  Issues  in  Science  and 
Technology  Online,  Spring  1997. 

This  article  stresses  that  evolving  national  priorities  and  budget  constraints  call  for  a  new  approach  to 
federal  spending.  Corroborating  the  NAS  report,  Allocating  Federal  Funds  for  S&T,  the  article  calls 
for  the  development  and  use  of  a  federal  S&T  budget.  It  calls  on  the  OMB  and  OSTP  to  implement 
an  annual  FS&T  analysis  as  a  part  of  the  normal  budget  review.  The  authors  state  that  using  this 
analysis  would  help  the  most  productive  programs  under  a  tight  budget  while  strengthening  the  case 
for  making  larger  investments  in  R&D. 


Teich,  Albert,  “Choosing  Among  Disciplines,”  AAAS  Science  and  Technology  Policy  Yearbook  1991,  Washington, 
D.C.:  American  Association  for  the  Advancement  of  Science,  1991,  pp.  41-45 

Teich  discusses  the  conflicting  perceptions  that  have  arisen  as  a  result  of  priority-setting  discussions. 

The  article  addresses  (1)  who  should  do  the  priority  setting,  (2)  who  would  use  the  results  and  how, 
and  (3)  what  the  outcomes  of  the  process  would  be.  The  articles  also  states  that  despite  the  concerns 
that  have  arisen,  the  budget  process  would  benefit  from  the  change.  Furthermore,  incorporating 
priority-setting  methods  based  on  technological  merit,  scientific  merit  and  social  merit  would  greatly 
improve  the  process. 


The  Government  Performance  and  Results  Act, YD.  101-189  andP.L.  100-456 


J 

26 


Require  the  Department  of  Defense  (DOD)  and  the  Office  of  Science  and  Technology  Policy  (OSTP) 
to  identify  priorities  for  critical  dual-use  technologies  for  national  security  and  economic  prosperity. 
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